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Abstract 

Building on Shulruf, Hattie and Tumen (2008), this work examines the capacity of various National 
Certificate in Educational Achievement (NCEA)-derived models to predict first-year performance in 
Biological Sciences at a New Zealand university. We compared three models: (1) the ‘best-80’ 
indicator as used by several New Zealand universities as a predictor of grade point average (GPA); (2) 
the ‘best-80’ as a predictor of outcome grade in biology courses and (3) ‘domain status’ in biology and 
chemistry as a predictor of outcome grade. These models span quantity, quality and competency 
measures in examining student performance and success at both university-wide and specific 
disciplinary levels. Results show that the models explain between 25 and 45% of the variance, which 
may present challenges if one of these models were to become the sole detemiinant for enrolment and 
limitation of entry policies, but can be useful in an advisory capacity. 

Keywords: NCEA, academic advising, biology, first year perfomiance, academic preparation for 
university studies, best-80 


Introduction 

This article investigates aspects of the capacity of secondary school awarded 
university entrance standards to predict first-year university performance for New Zealand 
secondary school leavers. Identifying productive secondary pathways to university success is 
of vital interest to educational institutions, students and governmental funding bodies. While 
this research addresses two years of intake and university performance of students in a single 
subject discipline, the approach can be applied both with other intake groups and for entire 
enrollment cohorts. 

Employing secondary qualification scores and subsequent university performance for 
three first-year Biology courses, this research is intended to enhance our understanding of 
‘best outcomes’ for first-year students. Such research can be used to advise secondary 
students, parents and staff and to consider university enrolment guidelines and expectations 
for success. We first introduce the New Zealand secondary education system with respect to 
university qualification, then outline our data sources and work through corresponding 
analyses, and finally offer conclusions. 

This paper was first presented at the Annual Conference of the Australasian Association for Institutional 
Research, Has institutional research come of age in Australia? in Geelong, 10-12 November, 2010. 
Correspondence to'. Dr Erik Brogt, Academic Development Group, University of Canterbury, Private Bag 
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Journal of Institutional Research, 16(2), 42 - 53 . 


43 


Sector context 

As recently as 2008, New Zealand university funding for undergraduates was based on 
student numbers (Engler, 2010a). In this model, the more students enrolled at a given 
institution, the more funding it received for undergraduate education. However, as student 
enrolments continued to rise, governmental reviews resulted in changes in that funding 
model’s ‘unlimited nature’. Caps or enrolment limits were introduced, ensuring greater 
certainty to the national education budget. Despite these caps, in 2008 there remained ample 
places for students desiring to attend university. 

By 2010 the landscape had changed. Partly owing to the global economic downturn, 
2009 enrolments increased markedly. Yet these were not matched by additional funding 
(Todd, 2010). Rather, alongside enrolment caps the government announced additional 
funding requirements beginning in 2012. These will tie a proportion of funding to indicators 
that include course passing rates, retention, program completion and progress to higher 
degrees. Such changes have increased attention on the need for universities to provide well- 
supported learning environments that enhance student success. These shifts have also 
triggered national debates about national university entrance (UE) requirements (Binning, 
2010 ). 


Combined, the funding changes present significant challenges to the management of 
undergraduate intake. Many universities are now reaching or exceeding enrolment caps, 
producing unfunded places and triggering governmental reviews. Consequently, some 
universities have implemented new enrolment criteria and curtailed or cancelled summer 
courses. To date, three of eight universities in New Zealand have introduced enrolment 
requirements based upon students’ secondary school exit qualification rankings— 
requirements that move beyond UE standards as set by the National Certificate in Educational 
Achievement (NCEA). By 2011 approximately half of all university enrolments will be so 
determined (Laugesen, 2010). These restrictions rely on the premise that secondary school 
performance can accurately predict tertiary success. 

NCEA overview 

Beginning in 2002, New Zealand introduced a new principal secondary qualification, 
the National Certificate of Educational Achievement, leading to a 2005 initial intake of 
NCEA students at university. Standards-based and divided into three levels, NCEA 
assessments are typically spread across the final three secondary school years. Students select 
specific subject areas or domains (e.g., Biology, Chemistry, English, etc.) and their 
knowledge and skills within these are assessed by a range of national standards (performance 
objectives). Specific standards are worth a predetennined number of possible credits, and 
these can be awarded at three perfonnance levels: ‘achieved’, ‘merit’, or ‘excellence’. Yet 
irrespective of level of attainment, the number of credits a student is awarded for a particular 
standard is identical. In the consideration of NCEA perfonnance for national standard for 
university entrance, both a student awarded ‘excellence’ and a student earning ‘achieved’ 
receive the same number of university intake credit points. 

Currently, students must achieve a requisite number of credits from an approved list of 
subjects to attain the national university entrance (UE) standard. As of 2010, the New Zealand 
requirements for UE mandate that a student earn 42 credits at NCEA level 3 or higher. Of 
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these 42 credits, at least 14 credits must be attained in each of two domains (subject areas), 
with the additional 14 credits earned from no more than two other domains. 

Data 


Two independent variables and two dependent variables have been used in the 
analysis contained within this article. The two independent (or predictor variables) are Best- 
80 rank scores and domain status. The dependent variables are students’ grade point average 
(GPA) and outcome grades awarded in the papers of interest. A brief definition and the 
derivation of each are outlined below. 

The Best-80 rank scores model 

The ‘Best-80 rank scores model’ is currently being employed for admissions 
consideration at the University of Auckland and other New Zealand universities. In the 
calculation of a Best-80 score for a student, the following process is employed (Shulruf, 
Hattie, & Tumen, 2008): 

• Select the best five domains from the list of university-approved domains. 

• Per domain, only 24 credits can be counted toward Best-80 score; in practice, most 
domains have a maximum of 24 achievement standards, but a variable number of unit 
standards. 

• Credits awarded with ‘excellence’ have a weight of 4, ‘merit’ a weight of 3, ‘achieved’ 
of 2 and unit standards of 2 (if applicable). 

• A maximum of 80 credits out of the five domains are counted toward the Best-80 score. 

Calculating Best-80 scores is accomplished through a software-automated process that selects 
each domain for which a student has credits (out of a potential 40 ‘university-approved’ 
subjects) and then calculates the Best-80 score for each possible combination. The maximum 
score is then used as each individual student’s Best-80 score. Because university entrance 
requires only two domains of 14 credits of university-approved standards, the theoretical 
range of Best-80 is 56-320 (derived from a possible minimum 28 credits at ‘achieved’ level 
to possible maximum 80 credits at ‘excellence’ level). 

Domain status in science-specific subjects 

Currently, full domain status indicates whether students have received at least 14 
credits at NCEA Level 3 with a performance level of ‘achieved’ or higher in the given 
domain. This aligns with the requirements to be awarded UE via NCEA. Students who have 
used some credits to make up part of their third domain are not considered as having domain 
status in the following discussion. 

Outcome grades 

Student records provide a means to examine first-year performance. Yet, as with 
regard to many majors, this undertaking is complicated since some students have enrolled in 
more than one of the three possible Biology courses and many take all three. Furthermore, 
some students may have enrolled in the same course in consecutive years; in such cases, only 
the first attempt (i.e., the initial enrolment grade) was used to represent performance. This 
university’s alphabetic scale marks from A+ to F were converted to numeric grades (Table 1). 
With respect to the cut-off percentages indicated (Table 1), these grades can be considered 
interval data (except with regard to tail-ends). A minor complication arising from grading 
policies at this university exists with the C- mark, which is locally regarded as a ‘non- 
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continuing pass’ and is rarely awarded (only two C- grades were assigned to NCEA-qualified 
students in this sample). 

Table 1 

Letter Grades Awarded at the University of Canterbury’s School of Biological Sciences Based 
on the Cut-Off Percentages Indicated, and the Numeric Grade Scale They Were Converted to 
for This Analysis 


Letter grade 

A+ 

A 

A- 

B+ 

B 

B- 

C+ 

C 

C- 

D 

F 

Numeric grade 

11 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

Cut-off percentage 

85 

80 

75 

70 

65 

60 

55 

50 

48 

40 

<40 


Grade point average (GPA) 

Grade point average (GPA) is also used as a dependent variable. GPAs are generated 
by the university. They are based on the grades a student is awarded and are weighted by the 
number of credits each course is worth. The GPA scale ranges from -1 through to 9 
(converted to 1-11 above). 


Analysis 

In considering the combinations of variables and the levels of measurement of data, a 
variety of analysis methods have been employed. These are contained in Table 2 and detail 
the principal comparisons between secondary preparation and performance and first-year 
university perfonnance. ‘Pathway-specific’ refers to whether the data used as independent 
variable (or predictor) or dependent variable (or outcome) is specific to the disciplinary 
pathway of interest-Biology. For example, in A below neither Best-80 nor GPA are specific 
to Biology, whereas in the case of C both secondary pathway and outcome grade in Biology 
papers have been used in analysis and both are specific to the discipline of interest. 

Table 2 

Variables, Measurements, Analysis and Pathway Specificity of Tests 


Variables 

Level of 
measurement 

Analysis 

IV pathway- 
specific 

DV pathway- 
specific 

(A) Best-80 by GPA 

Interval x interval 

Correlation 

No 

No 

(B) Best-80 by outcome 

Interval x ordinal 

Rank order 

No 

Yes 

grade 


correlation 



(C) Domain status by 

Nominal x ordinal 

Chi Sq 

Yes 

Yes 

outcome grade 






Interval x ordinal 

ANCOVA 

Yes 

Yes 


(A) Best-80 vs. GPA (overall GPA for all courses)-Is there a relationship between the Best- 
80 credits secondary students are a warded and their overall performance in first-year 
university? 

In examining whether secondary school preparation—and specifically, that 
preparation more directed towards university entrance standards—is associated with first-year 
university perfonnance, we have considered two years of NCEA intake and first-year GPA 



Journal of Institutional Research, 16(2), 42 - 53 . 


46 


for students enrolled in the three Biology courses. Table 3 contains correlation statistics for 
these students. As shown there is some association between GPA and Best-80, yet the 
strength of that relationship is relatively weak. 

To determine if student performance in Biology across two enrolment years could be 
considered equivalent with respect to overall preparation at secondary level, an independent 
sample t test was conducted on the mean number of total credits students undertook in 2007 
and 2008. Since no significant differences between the two mean scores ( df= 446, t = 1.3 6,p 
= .32) arose, the datasets from the two years were merged for the following analysis. 

Table 3 

Best-80 and First-Year GPA (all Courses) Correlation 


Year 

N (cases) 

Pearson’s R 

R 2 

P 

2007 

239 

.65 

.43 

<.001 

2008 

246 

.56 

.31 

<.001 


During these two years, neither the curriculum nor the assessments were substantially 
changed. Consequently, the grades in the two years are considered as equivalent, with 
differences in outcome grades attributable to the students (rather than any variation in 
assessment). When both years of enrolments are combined, N = 485, R = 37, r = .60, and 
p < .001. The scatter plot in Figure 1 illustrates the spread of secondary preparation results 
and first-year university performance for the combined grouping of students. 


Best 80 credits by GPA for students enrolled in 111 -113 during 

2007/2008 



Figure 1 

Scatter plot of 'Best-80 ’ by GPA for all 2007 and 2008 students enrolled in BIOL 111, 112 
and 113 
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(B) Best-80 vs. outcome grade-Is there a (stronger) relationship between the Best-80 
credits secondary students are a warded and their performance in specific courses in first- 
year university? 

This step of analysis tests whether Best-80 is a better predictor of performance in 
specific papers (in this case BIOL 111-113, where the dependent variable is pathway- 
specific) rather than overall perfonnance. Analysis uses Best-80 credits and outcome grades 
in the specific courses of BIOL111-113 to determine the strength of relationship. The test 
statistic, r s (Speannan’s rho) has been used because it is the appropriate measure of 
association that can accommodate ordinal level of measurement (individual grades). The 
Best-80 by outcome grade in the three courses (combined for each year) is shown in Table 4. 

Table 4 

Best-80 and Biology outcome grade correlations 


Course 

A (cases) 

Spearman’s 

R 2 

P 

BIOL 111 

341 

.57 

.32 

<.001 

BIOL 112 

272 

.51 

.26 

<.001 

BIOL 113 

271 

.53 

.28 

<.001 


The first observation to note is that, in contrast to predicting overall GPA, Best-80 is a 
poorer predictor of perfonnance in BIOL111-113. This is most likely due to the fact that 
GPA is a continuous variable whereas outcome grade is ordinal, resulting in more scatter in 
the latter case. 

(C) Pathways vs. outcome grades BIOL111,112,113 specific x specific measure-Is there a 
(stronger) relationship between particular pathways + performance at the secondary level 
and performance in associated courses in first-year tertiary? 

This third analysis factors the pathway-specific experiences for students in tenns of 
preparation for university (domain status) and the outcome in related courses (BIOL111-113) 
to detennine strength of relationship. In Table 5, outcome grades have been grouped into 
cohorts, ranging from F = failing grade through to A. The A, B and C grades include minus 
and plus as well as straight letter grades. Below each table section are the chi square test 
statistics and Speannan’s rho, using the convention r s < .2 as ‘weak’, .2 <r s < A as 
‘moderate’, and r s > A as ‘strong’ and p values. 
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Table 5 

NCEA Pathways and BI0L111 

-113 outcome grades 

BIOL111 



Domain 

Grade 

Has domain status 

Has no domain status 


F 

12 

5.3% 

17 

14.9% 

Biology 

C 

49 

21.6% 

29 

25.4% 


B 

89 

39.2% 

39 

34.2% 


A 

77 

33.9% 

29 

25.4% 


Total 

227 


114 




X 2 = H.02 

p = .012 

r s = .15 

p = .007 




Significant but weak 


BIOL111 

Domain 

Grade 

Has domain status 

Has no domain status 


F 

5 

2.8% 

24 

15.0% 

Chemistry 

C 

21 

11.6% 

57 

35.6% 


B 

68 

37.6% 

60 

37.5% 


A 

87 

48.1% 

19 

11.9% 


Total 

181 


160 




X= 72.17 

p = .000 

r s = .46 

p = .000 




Significant and strong 


Domain 

Grade 

Has domain status 

Has no domain status 


F 

21 

10.8% 

14 

17.9% 

Biology 

C 

39 

20.1% 

33 

42.3% 


B 

81 

41.8% 

22 

28.2% 


A 

53 

27.3% 

9 

11.5% 


Total 

194 


78 




/= 21.33 

p = .000 

r s = .26 

p = .000 




Significant and moderate 



F 

6 

5.6% 

29 

17.7% 

Chemistry 

C 

15 

13.9% 

57 

34.8% 


B 

43 

39.8% 

60 

36.6% 


A 

44 

40.7% 

18 

11.0% 


Total 

108 


164 




X 2 -43.64 

p = .000 

II 

p = .000 




Significant and moderate 


BIOL113 

Domain 

Grade 

Has domain status 

Has no domain status 


F 

12 

6.4% 

18 

21.7% 

Biology 

C 

42 

22.3% 

20 

24.1% 


B 

95 

50.5% 

36 

43.4% 


A 

39 

20.7% 

9 

10.8% 


Total 

188 


83 




X 2 = 16.06 

p = .001 

r s = .20 

p= .001 




Significant but weak 



F 

1 

1.0% 

29 

16.6% 

Chemistry 

C 

10 

10.4% 

52 

29.7% 


B 

51 

53.1% 

80 

45.5% 


A 

34 

35.4% 

14 

8.0% 


Total 

96 


175 




X 2 = 50.61 

p = .000 

r s = -43 

p = .000 




Significant and strong 
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Apparent from the previous table is the importance of Chemistry for student success 
across all three papers. For BIOL 111 and 113, having domain status in secondary Chemistry 
significantly improves chances of a student earning B or A grades in first year, and to a lesser 
extent the same holds true in BIOL 112. Students who undertake secondary Chemistry may 
well be better prepared to engage in the first-year Biology curriculum. With respect to those 
students who have domain status in secondary Biology a similar but lesser effect is observed. 
The Biology pathway has a weak relationship with outcome in 111 and 113 and only a 
moderate predictive capacity for 112. 

Yet this analysis at best shows a relationship of association-it does not indicate 
causation, and there may well be other effects involved (see also Engler, 2010b, pp. 4-5). For 
example, students who take Chemistry may be generally brighter than students who take 
Biology. Likewise there may be hidden effects from the pathway combination students take. 
That is, students who take Chemistry and Biology at the secondary level may be different to 
those that take only one or the other, as the remainder analysis examines. 

ANCOVA analysis 

To ascertain if domain status is a detennining factor in the outcome grade, it is 
necessary to control for the number of credits a student takes. This was done in a two-step 
process. In the first step we examined the linear relationships of the Best-80 score and the 
outcome grade for the four different groups (summarised in Table 6). 

Table 6 

Pearson Correlation Coefficients Between Outcome Grade in the Biology Courses and Best- 
80 for the Four Groups of Students Varying in Domain Status 


Course 

Chemistry + 
Biology 

Chemistry only 

Biology 

only 

Neither 

All students 

BIOL 111 grade 

.608* 

.456* 

.381* 

.054 

.565* 

BIOL 112 grade 

.541* 

.247 

.505* 

.015 

.526* 

BIOL 113 grade 

.722* 

.062 

.446* 

.125 

.528* 


*:p< .01 


The non-significant results for Chemistry only in BIOL112 and BIOL113 may be due to the 
small size of the group. It is also worth noting that these correlations are systematically higher 
than those found by Brogt, Sampson, Comer, Turnbull, and McIntosh (2011), who used the 
total number of NCEA credits rather than a Best-80 score. 

In the second step, groups that had significant relationships between outcome grade 
and Best-80 score for a course were examined using ANCOVA analysis, with group 
membership as a factor and the Best-80 score as the covariate. An interaction between the 
Best-80 score and group membership was built into the model to test if the slopes of the 
relationships between Best-80 score and grade were homogeneous. Cases where the 
interaction was not statistically significant (at the .05 level) were removed from the model, 
allowing the variance to be absorbed into the main effects. In addition, a lack of fit test 
showed non-significant results for all three courses, meaning that a linear fit is acceptable. 

The results of the ANCOVA analysis for each course are described in the following section. 
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BIOL111 

For BIOL 111 the total credits covariate was significantly related to student outcome 
grades (Table 7). After controlling for the covariate, group membership was found to be 
significant as well (Table 8). The pair-wise comparisons between the groups showed that 
students who had taken Biology and Chemistry did significantly better than those who had 
taken Biology only. However, no significant differences were found between Biology and 
Chemistry vis-a-vis Chemistry only, or between Chemistry only and Biology only. The 
overall corrected models explained around 38% of the observed variance. 


Table 7 

Summary Statistics for ANCOVA Snalysis ofBIOLlll 


Source 

if 

F 

P 

partial ri 2 

Best-80 

1 

91.5 

<.001 

.247 

Intercept 

1 

24.5 

.011 

.023 

Group membership 

2 

5.4 

.002 

.044 

Error 

279 




Corrected model 

3 

56.3 

<.001 

.377 


Table 8 

Pair-Wise Comparisons Between Groups for BIOL111 


Group 1 

Group 2 

Estimated mean 
group 1* 

Estimated mean 
group 2* 

Difference 

Standard 
error of the 
difference 

P 

Chemistry + 

Chemistry 

7.78 

7.26 

.52 

.31 

.098 

Biology 
Chemistry + 

Biology 

7.78 

6.77 

1.01 

.29 

.001 

Biology 

Chemistry 

Biology 

7.26 

6.77 

.49 

.35 

.160 


* Means are calculated based on a Best-80 score of 212.33 

BIOL112 

The total credits covariate was also significantly related to student outcome grades in 
BIOL 112 (Table 9). However, after controlling for the covariate, group membership was not 
a statistically significant predictor for outcome grade. The overall corrected model explained 
about 36% of the observed variance. 


Table 9 

Summary Statistics for ANCOVA Analysis ofBIOL112 


Source 

df 

F 

P 

partial r| 2 

Best-80 

1 

71.1 

<.001 

.271 

Intercept 

1 

.062 

.804 

<001 

Group membership 

1 

2.6 

.112 

.013 

Error 

191 




Corrected model 

2 

42.2 

<.001 

.362 
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BIOL113 

The total credits covariate was significantly related to student outcome grades again in 
BIOL 113 (Table 10). Similar to BIOL 111, after controlling for the covariate, group 
membership also had a significant effect on grades in BIOL 113 (Table 11). The overall 
corrected models explained around 45% of the observed variance. The interesting feature in 
this model is the significant interaction of Best-80 score * group membership. This means that 
the slopes of the two regression lines are significantly different. We interpret this to indicate 
that in the case of BIOL 113 having Chemistry as an NCEA domain has more impact on first- 
year student outcomes than having other subjects. The interaction highlights the importance 
of Chemistry in Biology. 


Table 10 

Summary statistics for ANCOVA analysis ofBIOL113 


Source 

df 

F 

P 

partial ri 2 

Best-80 

1 

86.1 

<.001 

.172 

Intercept 

1 

1.6 

.215 

.008 

Group membership 

1 

1.6 

.211 

.008 

Best-80 * Group membership 

1 

4.2 

.041 

.022 

Error 

184 




Corrected model 

3 

49.4 

<.001 

.446 


Table 11 

Pair-Wise Comparisons Between Groups for BIOL113 




Estimated mean 

Estimated mean 


Standard error 
of the 

Group 1 

Group 2 

group 1* 

group 2* 

Difference 

difference p 

Chemistry + 
Biology 

Biology 

7.05 

6.14 

.91 

.28 .001 


* Means are calculated based on a Best-80 score of 201.03 

Discussion 

Alongside similar Australian research (Green, Brown, & Ward, 2009), this analysis 
indicates that, even in the cases of apparently straightforward disciplinary pathways from 
secondary school, predicting first-year student success is challenging, with unexpected 
results. For example, for Biology students at university domain status in secondary school 
Chemistry holds greater predictability for first-year success than other domains. In 
considering ‘depth’ versus ‘breadth’ of studies at secondary level, such findings may illustrate 
differences with research from other sectors (Schwartz, Sadler, & Tai, 2008) that notes 
possible counterproductive consequences for secondary ‘breadth’ with potential university 
Biology students. 

Shulruf, Hattie and Tumen (2008) argue for a shift in emphasis on intake to quality, 
with university admissions based up ‘higher achievement in fewer credits’. In this data 
sample, mandating that students earn 18 credits of NCEA level 3 Biology and 18 of 
Chemistry (vs. 14 for domain status currently) could increase their chances of success in two 
of the first-year courses. Yet nearly 40% of current students would be excluded through such 
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requirements. While the mean first-year passing rate of an overall intake can be closely 
predicted (Scott, 2008), individual passing rates are less easily predicted, even in the case of 
relatively straightforward disciplinary pathways. 

Conclusions 

All stakeholders involved—including students, secondary schools, universities and 
national policymakers—desire clear and evidence-based secondary pathways to university 
entrance and, preferably, achievement. Yet basing admissions and predicting individual 
success on the basis of NCEA secondary perfonnance remains a problematic exercise. For 
incoming students pursuing a university Biology curriculum, there is evidence for the 
importance of Chemistry as a required secondary school pathway. However, there is relatively 
low predictive capacity for first-year perfonnance even when NCEA domain status is 
achieved. Consequently, policies that institute blanket requirements based on secondary 
performance or specific NCEA pathways prior to university entrance may well deny 
enrolment to many students with a good chance of success. In the case of potential 
disciplinary pathway requirements, it should also be noted that some disciplines have 
reasonably clear pathways from secondary to tertiary (like Biology and Chemistry for 
Biological Sciences), whereas others do not (e.g., Psychology). If a clear secondary pathway 
can be identified, an analysis like the one presented in this study can be a useful tool for 
individual departments in advising secondary students as to the best preparation for tertiary 
success in that discipline. 
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